Benchmarking Music Information Retrieval Systems
نویسندگان
چکیده
Our goal is to create an accurate and effective benchmarking system for music information retrieval (MIR) systems. This will serve the multiple purposes of inspiring the MIR community to add additional features and increased speed into existing projects, and to measure the performance of their work and incorporate the ideas of other works. To date, there has been no systematic rigorous review of the field, and thus there is little knowledge of when an MIR implementation might fail in a real world setting. Benchmarking MIR systems is currently hindered by the diversity of the systems, by their relatively new and unrefined nature, and by the limited number of accessible systems. Thus most of what will be described here will be introductory and will lay down the framework for future benchmarking and analysis. Particular attention will be paid to the evaluation issues surrounding retrieval of audio in test collections.
منابع مشابه
MIR Benchmarking: Lessons Learned from the Multimedia Community
Music Information Retrieval may be perceived as part of the larger Multimedia Information Retrieval research area. However, many researchers in Music Information Retrieval are unaware that the problems they deal with have analogous problems in image and video retrieval. Many issues concerning the creation of testbed digital libraries and effective benchmarking of information retrieval systems a...
متن کاملPrototyping a Vibrato-Aware Query-By-Humming (QBH) Music Information Retrieval System for Mobile Communication Devices: Case of Chromatic Harmonica
Background and Aim: The current research aims at prototyping query-by-humming music information retrieval systems for smart phones. Methods: This multi-method research follows simulation technique from mixed models of the operations research methodology, and the documentary research method, simultaneously. Two chromatic harmonica albums comprised the research population. To achieve the purpose ...
متن کاملBeyond Recall and Precision: A Full Framework for MIR System Evaluation
The Cranfield tests are perhaps the most well-known and often cited example of benchmarking of information retrieval systems. However, of the six criteria that Cleverdon identified as pertinent for analysis of information retrieval systems, only two, precision and recall, are typically investigated. We argue that the other criteria are also vitally important for advanced IR systems such as a mu...
متن کاملMusiClef: Multimodal Music Tagging Task
MusiClef is a multimodal music benchmarking initiative that will be running a MediaEval 2012 Brave New Task on Multimodal Music Tagging. This paper describes the setup of this task, showing how it complements existing benchmarking initiatives and fosters less explored methodological directions in Music Information Retrieval. MusiClef deals with a concrete use case, encourages multimodal approac...
متن کاملOverview of MusiCLEF 2011
MusiCLEF is a novel benchmarking activity that aims at promoting the development of new methodologies for music access and retrieval on real public music collections. A major focus is given to multimodal retrieval that, in the case of music collections, can be obtained by combining content-based information, automatically extracted from music files, with contextual information, provided by user...
متن کامل